HighLight: Using a Log-structured File System for Tertiary Storage Management

Log-structured file systems (LFSs) were developed to eliminate latencies involved in accessing disk devices, but their sequential write patterns also match well with tertiary storage characteristics. Unfortunately, existing versions only manage memory caches and disks, and do not support a broader storage hierarchy. Robotic storage devices offer huge storage capacity at a low cost per byte, but with large access times. Integrating these devices into the storage hierarchy presents a challenge to file system designers. HighLight extends 4.4BSD LFS to incorporate both secondary storage devices (disks) and tertiary storage devices (such as robotic tape jukeboxes), providing a hierarchy within the file system that does not require any application support. This report presents the design of HighLight, proposes various policies for automatic migration of file data between the hierarchy levels, and presents initial migration mechanism performance figures. Log-structured file systems, with their sequential write patterns and large transfer sizes, extend naturally to encompass a storage hierarchy. HighLight is flexible enough to support a variety of potential migration policies. When data are disk-resident, access is nearly as fast as access to files in the base 4.4BSD LFS. When disk arm contention is absent, transfers to tertiary devices can run at nearly the tertiary device transfer speed. 1 1This report is substantially derived from an article originally published in the USENIX Association Conference Proceedings, January 1993. Permission has been granted by the USENIX Association to incorporate portions of that paper in other publications. Portions of this report Copyright c USENIX Association, 1993. This research was sponsored in part by the University of California and Digital Equipment Corporation under Digital’s flagship research project “Sequoia 2000: Large Capacity Object Servers to Support Global Change Research.” Other industrial and government partners include the California Department of Water Resources, United States Geological Survey, MCI, ESL, Hewlett Packard, RSI, SAIC, PictureTel, Metrum Information Storage, and Hughes Aircraft Corporation. This work was also supported in part by Digital Equipment Corporation’s Graduate Engineering Education Program.